[00:05.020 --> 00:08.720]  Hello and welcome to Attacking and Defending Blockchain Nodes,
[00:08.720 --> 00:13.820]  where we're going to learn about a variety of risks and mitigations that we can implement
[00:13.820 --> 00:18.560]  to defend a very core component of the blockchain systems.
[00:20.000 --> 00:25.040]  We'll try to do a live discussion during the talk while we ask a question, so feel free to join
[00:25.900 --> 00:33.080]  DEF CON's Discord on BCV General Tech's channel, and I'll try to ask questions and respond live.
[00:34.380 --> 00:39.540]  In case we haven't met, my name is Peter Kaczynski. I'm a blockchain security engineer at
[00:39.540 --> 00:45.380]  Coinbase, where I spend most of my time breaking a variety of different blockchain systems and
[00:45.380 --> 00:51.420]  smart contracts. I'm also a writer for the Blockchain Threat Intelligence Newsletter,
[00:51.420 --> 00:59.060]  where I try to cover all the latest news, events, hacks, scams in the blockchain security ecosystem.
[00:59.840 --> 01:04.700]  Last year, you may have also participated in the Capture the Coin competition,
[01:04.700 --> 01:11.500]  which I helped organize. My background before deep diving into the cryptocurrency world
[01:11.500 --> 01:16.180]  was as a malware reverse engineer at FireEye, where I looked at a whole lot of APT malware
[01:16.180 --> 01:22.040]  and a penetration tester for the Federal Reserve System, where I was breaking Finance 1.0.
[01:24.120 --> 01:30.300]  So, just some idea, give you an idea of what we're going to talk about. I'm not going to drop
[01:30.300 --> 01:36.520]  any old days today. What my goal here is to educate, to share knowledge, to also learn from you.
[01:37.180 --> 01:42.420]  With the goal of just making, discussing, like, how can we secure node infrastructure?
[01:43.080 --> 01:48.700]  So we'll start with a simple exercise on how do we know, what is the state of truth that tells us
[01:48.700 --> 01:55.300]  how many coins we actually have? We'll go through a variety of examples of how nodes break, and
[01:55.300 --> 02:01.280]  we'll discuss a few of the ways that they could break in the future. Finally, I'll introduce a
[02:01.280 --> 02:09.260]  solid node security threat model for generic nodes, which you can apply later on for a specific node
[02:09.260 --> 02:15.300]  implementation. And most importantly, discuss a top 10 list of security defenses that you can
[02:15.300 --> 02:19.960]  implement today, whether you're a standalone operator, or you're working for an exchange,
[02:19.960 --> 02:28.280]  and share lessons learned from trying to secure nodes of my own. So with that, let's talk about
[02:28.280 --> 02:34.340]  how many coins you have. So what is, if you're running, if you wanted to, at any given moment,
[02:34.340 --> 02:40.000]  tell me how many coins of any given asset that you have, how do you go about that?
[02:40.660 --> 02:45.300]  So, well, you'll probably have a piece of software on your computer, on your phone,
[02:45.300 --> 02:51.840]  or a hardware wallet, maybe, which has some specific logic dedicated just for a specific
[02:51.840 --> 02:56.640]  asset. So if you have a Bitcoin wallet, or Ethereum, or Monero, or some other coin,
[02:56.640 --> 03:03.060]  sorry if I didn't list here. This particular wallet communicates with a wider network,
[03:03.060 --> 03:09.020]  somewhere to the cloud, and asks a question, hey, how much coins do I have? What do I actually own?
[03:09.460 --> 03:14.000]  In case of Bitcoins, you try to enumerate how many outputs that you have unspent.
[03:14.440 --> 03:19.000]  Ethereum is different in the sense that it's an account-based system, so you have to look up
[03:19.000 --> 03:26.880]  how much of the asset that I actually have on the network. The thing is, it gets a little bit
[03:26.880 --> 03:32.060]  more complicated. So if we start getting a little bit more granular, we're going to see that,
[03:32.060 --> 03:37.360]  in fact, we don't communicate with just some unknown entity, abstract entity. We actually
[03:37.360 --> 03:43.140]  communicate to specific nodes for that specific coin. So Bitcoin wallet communicates to Bitcoin
[03:43.140 --> 03:50.740]  nodes, Ethereum to Ethereum, Monero to Monero, and so on. This introduces some complications,
[03:50.740 --> 03:57.800]  because we realize that there are a lot of third-party components involved. So if we just
[03:57.800 --> 04:03.880]  want to manage our assets, we have to communicate to a whole bunch of software which is maintained
[04:03.880 --> 04:11.480]  by third parties. So just to give you like a few more very simplistic threat model here,
[04:11.480 --> 04:16.900]  let's talk about components of a friend's coin, just the imaginary coin from the Mario friend's
[04:16.900 --> 04:24.900]  land. So you have your wallet communicating to a node, communicating to the overall gossip network
[04:24.900 --> 04:33.000]  of some sort, trying to propagate blocks. You also have your developers who are contributing to the
[04:33.000 --> 04:40.140]  GitHub repository, so you have to rely on maintaining the repository, compiling the node maybe, or
[04:40.140 --> 04:47.640]  downloading binaries there, and so on. And this all works really well, and until we start thinking
[04:47.640 --> 04:54.420]  that, well, how can we be sure that all the software libraries, all the standalone software
[04:54.420 --> 05:00.940]  infrastructure does not act maliciously, does not drop, modify, or craft transactions to steal your
[05:00.940 --> 05:07.120]  funds. So let's see how this could happen. So we have this Mr. Robot character trying to attack
[05:08.060 --> 05:13.380]  a variety of different points of this threat model. So we'll start left to right,
[05:13.380 --> 05:19.040]  so we can see that we can attack the network connection between your node and the
[05:19.040 --> 05:25.160]  cloud. So someone could start dropping blocks, try to eclipse attack your node, so basically to
[05:25.160 --> 05:31.680]  isolate it to make sure it's not communicating to the rest of the network properly. The same character
[05:31.680 --> 05:36.620]  can also attack your node directly if there are some unknown weaknesses that they can exploit to
[05:36.620 --> 05:41.360]  do some kind of code execution or crash it. They can prevent you from accessing the
[05:41.360 --> 05:48.540]  network. At last, we also have to think about developers. The bad actors out there, they can
[05:48.540 --> 05:56.300]  go after developers like the toad over there. They can make them commit bad code. And I'll
[05:56.300 --> 06:02.200]  cover a few real-world examples where this does happen. So the idea is that
[06:03.980 --> 06:09.580]  it's an invitation to start thinking about the node security as a holistic
[06:09.580 --> 06:13.100]  ecosystem. So there are a variety of different components which are involved.
[06:15.040 --> 06:20.320]  So the examples that I just mentioned, they're not theoretical, they're practical, they're based
[06:20.320 --> 06:26.900]  on real-world incidents. So for example, in May 2014, someone attacked the BGP network
[06:26.900 --> 06:34.220]  used to eclipse attack Doge and Bitcoin miner pools. So this was, in case you're not
[06:34.220 --> 06:42.680]  familiar with BGP hijacks, it's a way to effectively isolate whole large swaths of network blocks
[06:42.680 --> 06:47.920]  in order to make them route traffic to unexpected directions. This is something that happened
[06:48.400 --> 06:53.060]  a decade ago, I believe with YouTube, where all the traffic was redirected to
[06:53.920 --> 06:57.600]  to Pakistan, I believe. And it keeps happening again and again.
[06:59.180 --> 07:04.220]  This is a threat that we have not observed within cryptocurrency world for a while, but
[07:04.220 --> 07:08.700]  it's something that we have to be aware of and build threat models around.
[07:09.400 --> 07:15.100]  On the EOS side, after the release of the Node software, they published a bug bounty.
[07:15.300 --> 07:19.860]  And oh boy, did they receive a lot of good finds there, because one, there was immediately a code
[07:19.860 --> 07:25.560]  execution flaw that was discovered. And for a while at least, they kept on getting one
[07:25.560 --> 07:32.560]  after another of high-value bounties submitted to them, which included denial of service, code
[07:32.560 --> 07:40.640]  execution, and similar. Finally, in 2019, Monero website was compromised. The binaries for the
[07:40.640 --> 07:46.460]  wallet were backdoored to steal clients' funds. So this is an example of how we have to look out for
[07:47.140 --> 07:51.040]  the developer. Where are you getting that software? How trustworthy is that resource?
[07:51.040 --> 07:57.920]  What can you do to verify it? So this was just over the few years, but I just want to instill
[07:57.920 --> 08:02.600]  in you that this is an ongoing problem. These are all the different things happening this year.
[08:02.820 --> 08:08.340]  So in March, we had on the testnet, but we still had a flaw discovered in the way Solana was
[08:08.340 --> 08:13.900]  processing transactions, where it was not doing sufficient validation. So it resulted in 500
[08:13.900 --> 08:20.560]  million Solana getting stolen. Filecoin, again, had a nasty inflation bug, fortunately discovered
[08:20.560 --> 08:26.280]  on testnet, exploited also on testnet, but it resulted in 9 billion Filecoins minted,
[08:26.280 --> 08:30.820]  when only 2 billion could have existed in the first place. Inflation bugs are particularly nasty
[08:30.820 --> 08:36.160]  because these are the attacks not against individual users, but against the entire coin
[08:36.160 --> 08:43.280]  ecosystem. Tendermint is a library behind a variety of projects. It's a consensus mechanism used in
[08:43.280 --> 08:48.760]  projects like Cosmos, Oasis, and others. A denial of service vulnerability was discovered there
[08:48.760 --> 08:56.260]  when parsing specially crafted invalid blocks, which basically crashed the nodes. So if this
[08:56.260 --> 09:01.120]  was successfully executed in the wild, Cosmos was likely not vulnerable to this because they
[09:01.120 --> 09:06.260]  didn't upgrade just to that particular vulnerable version. But if that was the case and the
[09:06.260 --> 09:10.920]  vulnerability was maliciously discovered, then it would result in a network halt.
[09:11.520 --> 09:15.300]  The last example, and probably the most critical and interesting one,
[09:15.880 --> 09:22.580]  both from the impact and the way that it was handled, is Ravencoin. Ravencoin had an inflation
[09:22.580 --> 09:30.420]  bug discovered just last month in July. The inflation bug vulnerabilities basically allow
[09:30.420 --> 09:35.920]  someone to print money out of thin air, which is extremely dangerous because it allows an attacker
[09:35.920 --> 09:43.640]  just to profit and sell value to exchanges, and it basically impacts every single user of the coin.
[09:43.640 --> 09:47.840]  What's interesting about this particular vulnerability was that it was actually
[09:47.840 --> 09:54.980]  maliciously introduced. It was introduced back in January by an account which was seemingly
[09:54.980 --> 10:01.260]  making adjustments to the way the comments were made, on the debugging views were made.
[10:02.360 --> 10:07.880]  And the core developers unfortunately missed that until the point that someone reported them,
[10:08.320 --> 10:11.380]  a third party, which was building a blockchain explorer for Ravencoin,
[10:11.380 --> 10:14.400]  there's something weird going on in your network. We're observing someone
[10:15.060 --> 10:21.220]  minting a whole bunch of RVN and also, by the way, selling them on exchanges and profiting from it.
[10:21.600 --> 10:29.180]  So the Ravencoin team was able to issue an emergency patch, worked with the miners,
[10:29.180 --> 10:36.260]  but it's still 300 million RVNs were minted and exchanged for some other coins.
[10:36.780 --> 10:40.200]  Something that we have to learn from.
[10:42.140 --> 10:49.060]  So before we go into the section on how to secure the nodes, how to look out for those
[10:49.060 --> 10:53.660]  vulnerabilities, let's first think about all the different ways that we can break the nodes.
[10:54.460 --> 11:00.060]  So just to define the scope right away, we're not going to talk about the features of a node,
[11:00.060 --> 11:05.060]  which are documented and designs of the node. So if something implements proof of work,
[11:05.060 --> 11:12.020]  we're not going to deep dive into 51% attacks, reorgs, other protocol design issues. We'll accept
[11:12.020 --> 11:17.360]  them as is. What we're more concerned about is if the node promises to do something correctly,
[11:17.360 --> 11:21.560]  and it fails to do that, that's what we need to pay attention to.
[11:22.340 --> 11:30.640]  Key management is also outside of this. So you should probably be running nodes on a variety
[11:30.640 --> 11:34.620]  of tiers. So if you just want something which replicates the state of the network, you should
[11:34.620 --> 11:41.440]  probably keep your keys on a separate system. Incorrect usage. So I assume that if you're
[11:41.440 --> 11:45.240]  running a specific node that you read the documentation, you read everything that is
[11:45.240 --> 11:53.560]  known about it, so you don't necessarily make mistakes. Like a good example of that is Ripple's
[11:53.560 --> 12:00.380]  TF partial payment flag, where if you do not implement that particular flag correctly, then
[12:00.380 --> 12:06.080]  whenever someone sends you some payment with that flag set, it may appear that you actually
[12:06.080 --> 12:12.840]  receive those funds. But in reality, it's a IOU, essentially, that I will send you those things
[12:12.840 --> 12:17.420]  eventually, but not right now. So a lot of exchanges were exploited as a result of that.
[12:17.840 --> 12:22.740]  Finally, we're not going to focus on the layer two vulnerabilities such as smart contracts,
[12:22.740 --> 12:30.180]  DeFi, and so on. So what's in scope? What's in scope are the implementation flaws. So protocol,
[12:30.180 --> 12:37.740]  software flaws, attack resilience, can the node stand up to sustained attacks? Infrastructure. So
[12:37.740 --> 12:42.140]  we're not looking at nodes as just this one thing, and we're protecting just the node itself,
[12:42.140 --> 12:46.640]  and don't care about anything that it's built in software. We're worrying about underlying OS,
[12:46.640 --> 12:50.580]  we're worrying about the network stack, and anything that goes around it.
[12:51.080 --> 12:56.900]  Finally, management. Human factor is, always was, always will be the weakest link in the security
[12:56.900 --> 13:02.260]  chain. So we will discuss access management, we'll talk about configuration, source control,
[13:02.260 --> 13:11.240]  and all other ops type of things. So with that, just before we dive into the exercise, just wanted
[13:11.240 --> 13:16.660]  to establish some terminology so we're all on the same page. The threat model, and this is taken
[13:16.660 --> 13:21.840]  from OWASP's excellent threat modeling cheat sheet if you want to review it and build your own threat
[13:21.840 --> 13:30.100]  models. Threat model itself is a process to identify potential threats. So we want to enumerate
[13:30.100 --> 13:34.360]  all the different ways that things can go wrong, we can assess those threats, what is the risk
[13:34.360 --> 13:38.960]  coming from them? Are they basically going to result in me losing money, or is it just
[13:39.340 --> 13:47.500]  a denial of service, or some other maybe less critical impact? And also later on, we'll use
[13:47.500 --> 13:54.320]  that to prioritize mitigations. Why would we spend hours or weeks trying to address a risk
[13:54.320 --> 14:00.340]  which is not really that impactful? An important thing that we're going to do next, we'll define
[14:00.340 --> 14:06.020]  the attack surface. An attack surface, think of it as all the different ways that someone can access,
[14:06.020 --> 14:12.640]  can attack your node or node infrastructure, and the threat agents, which are the entities
[14:12.640 --> 14:20.040]  which have enough skill, opportunity, and interest to attack your nodes.
[14:20.960 --> 14:27.960]  So let's begin with just a really basic system, it's not interacting with any other nodes, it's
[14:27.960 --> 14:34.580]  not managed by anything, just like a really basic core software. So we have the node itself, which
[14:34.580 --> 14:43.720]  all it does is it's processing the packets, it's processing the transactions, blocks, does
[14:43.720 --> 14:49.080]  verification, it stores the blocks in some kind of storage system, it parses configuration files
[14:49.080 --> 14:55.360]  so it knows how to properly look at those things, and it may also include a VM interpreter process
[14:55.360 --> 15:01.020]  where it can interpret smart contracts or anything like full-blown, Turing-complete,
[15:01.020 --> 15:05.600]  Ethereum-style smart contracts, or something a little bit more locked down like Bitcoin script.
[15:07.080 --> 15:13.460]  Now, let's add some more components to that. In order for node to be useful, it needs to
[15:13.460 --> 15:18.060]  have some kind of management interface, and most nodes implement some kind of RPCE
[15:18.620 --> 15:24.100]  network client, so we have to think about the threats coming from that, who can access this
[15:24.100 --> 15:29.520]  thing, what is it they can do, and of course, node administrators themselves who are managing
[15:29.520 --> 15:33.840]  the machine, they can shut them down, they can bring it back online, they can reconfigure them
[15:33.840 --> 15:39.900]  and maybe introduce some flaws that I will cover in a moment. And also, these are just humans,
[15:39.900 --> 15:47.260]  can they be attacked on their own with malware or maybe any other social engineering? Finally,
[15:47.260 --> 15:51.040]  nodes need to communicate to other nodes to figure out what is the state of the network,
[15:51.040 --> 15:55.340]  and this is where a whole bunch of different things are coming into play, like network
[15:55.340 --> 16:00.760]  infrastructure, other nodes that we need to talk to, and miners. So network infrastructure
[16:00.760 --> 16:06.820]  includes basically underlying layers of communication for the nodes, so think of BGP
[16:08.220 --> 16:15.120]  and DNS client, DNS services, and just basic routing like TCP connections, UDP,
[16:15.120 --> 16:21.900]  all of that is in scope and needs to be secured as well. Finally, nodes are just not appearing
[16:21.900 --> 16:26.280]  on your machines out of nowhere, they must come from somewhere. Unless you are the developer,
[16:26.280 --> 16:31.460]  you probably are relying on third-party repositories, which also in turn rely on
[16:31.460 --> 16:39.660]  third-party dependencies. So this is something that we need to be aware of, because as I mentioned
[16:39.660 --> 16:46.440]  with Ravencoin, someone was able to backdoor the assets through a very sneaky code change
[16:46.440 --> 16:51.760]  and something that we also need to consider within our threat model.
[16:53.540 --> 16:58.540]  So with that in mind, join the BCV General Text on the DEF CON Discord channel
[16:59.600 --> 17:05.180]  and I'll ask you questions to just brainstorm together on how we can break nodes.
[17:06.760 --> 17:11.120]  So the first thing we're going to attack is the core. This is the node software
[17:11.120 --> 17:21.000]  itself. This is what processes the blocks, the transactions, and needs to verify them correctly.
[17:21.420 --> 17:26.840]  If you are this sneaky Mr. Robot character and you wanted to attack this,
[17:26.840 --> 17:31.320]  how would you do it? So I'll look in the channel for your responses.
[17:39.860 --> 17:46.900]  So I have one comment of software supply chain attack. So this is something that we're going to
[17:46.900 --> 17:52.300]  cover in the software repository attack, but I'm talking about the core itself. That's the
[17:52.300 --> 18:02.060]  piece of software that does the actual blockchain logic. BGP attacks to create an eclipse.
[18:03.160 --> 18:10.300]  So again, so this is something that we have to worry about on the network connectivity side.
[18:16.190 --> 18:22.740]  Let's see...
[18:23.060 --> 18:30.280]  Yeah, so a denial of service against the nodes to crash with the bad blocks, malformed block headers.
[18:30.300 --> 18:38.080]  So that's exactly that. So we can craft a packet, think of a tendermint flaw,
[18:38.080 --> 18:46.540]  which basically crashes a node or maybe executes some kind of really nasty code execution attack.
[18:47.080 --> 18:52.700]  So just to give you some background, this is how I created the model. So there's a title,
[18:52.700 --> 18:58.840]  protocol vulnerabilities, severity. So if you can crash or do code execution, know the severity is
[18:58.840 --> 19:05.380]  high. Probability, I would say medium. It's just harder to find, harder to exploit. It requires
[19:05.380 --> 19:11.860]  dedicated time to discover those things. But the impact is that anything that is not properly
[19:11.860 --> 19:17.380]  processing blocks, transactions, what is the correct fork, messing with the governance and so
[19:17.380 --> 19:21.860]  on, inflation vulnerabilities, this is all in scope for this. And it will have a severe impact
[19:22.600 --> 19:30.560]  on the node. However, I want you to consider it's less sexy of an attack. So we're not trying to
[19:30.560 --> 19:35.700]  find some kind of ode within blocks and transactions. And consider that nodes actually
[19:35.700 --> 19:42.260]  run all sorts of third-party software. So for example, Parity Nodes, they run or OpenTheorem
[19:42.260 --> 19:50.640]  now, they run web UIs to manage the thing. A bunch of node software relies on third-party
[19:51.220 --> 19:57.720]  database libraries and standalone databases to store blocks. So all of that are fair game to
[19:57.720 --> 20:04.440]  attack as well. So if you are Mr. Robot and you wanted to attack your node infrastructure,
[20:04.440 --> 20:10.740]  I'm not going to spend time trying to find old days for the way that you parse packets. It's a
[20:10.740 --> 20:16.260]  lot easier for me to find out what is the weakest link, what is the common off-the-shelf software
[20:16.260 --> 20:21.300]  that you're running, such as if you're running Nginx and for some reason it's a vulnerable
[20:21.300 --> 20:26.500]  version, I'll try to attack that as well. I'll try to attack that instead because it probably has
[20:26.500 --> 20:32.200]  more eyes on it and more vulnerabilities discovered so far. So consider those as your first line
[20:32.200 --> 20:37.420]  of things to lock down and the probability is definitely higher than finding a very
[20:37.420 --> 20:43.460]  directed attack to like in the way you parse blocks. So let's continue in the same
[20:44.840 --> 20:49.560]  style. So software repository attacks. So someone in the channel already mentioned that,
[20:49.560 --> 20:55.480]  you know, the supply chain attack. So whether you're attacking the software repository itself,
[20:55.480 --> 21:01.140]  so you can, you know, like a Ravencoin example or Monero example, if you can compromise the
[21:01.140 --> 21:09.660]  software repository itself, then you can introduce arbitrary changes, which are maybe
[21:09.660 --> 21:14.180]  different degrees of how hidden they are, depending how many eyes are looking at the repository.
[21:15.460 --> 21:22.760]  And again, this can result in fund theft or money printing type of vulnerabilities. However,
[21:22.760 --> 21:28.980]  what I want to invite you to also consider is not just the main software repository, but also
[21:28.980 --> 21:34.560]  software dependencies. So an example, not as actually a node vulnerability, but I believe it
[21:34.560 --> 21:43.100]  was Trinity Wallet, which had a, it was backdoored on the source code level. But the way that it was
[21:43.100 --> 21:47.500]  backdoored was not through the main repository, but through one of the dependencies. So someone
[21:47.500 --> 21:53.040]  figured out what is the weakest link in the supply chain to that particular piece of software,
[21:53.040 --> 21:59.200]  backdoored that, and then they were able to introduce code that was stealing keys.
[21:59.780 --> 22:04.800]  So again, probability is high on that as well, and severity is also extremely high.
[22:07.240 --> 22:10.640]  I mentioned this before, you know, human factor is always the weakest
[22:10.640 --> 22:14.520]  one. So node administrator, how can we attack that?
[22:17.760 --> 22:28.720]  Yep, phishing, send phishing email with some kind of malware. You know, the impact is very high.
[22:28.860 --> 22:36.440]  They can reconfigure the node, they can introduce a set of nodes, which they essentially creates
[22:36.440 --> 22:40.460]  an eclipse attack where you only communicate to something that is controlled by the attacker.
[22:40.680 --> 22:45.300]  Certain blockchains, they have a switch you can flip, which basically allows it to accept all
[22:45.820 --> 22:51.080]  blocks and transactions without doing any verifications, like a debugging mode almost.
[22:51.120 --> 22:54.840]  There's a variety of things that administrator can do and the variety of ways that administrator
[22:54.840 --> 22:58.760]  can get attacked through phishing, malware, social engineering, you name it.
[23:00.440 --> 23:05.680]  The last, you get the gist of how we're going through different components. The last one I
[23:05.680 --> 23:12.340]  want to cover is the network attack. So, correct, so BGP attacks to create an eclipse.
[23:13.400 --> 23:23.020]  So, exploiting the network infrastructure, you know, all you know, a lot of nodes,
[23:23.020 --> 23:26.800]  the way that they're built, is that they need to first boot up. When they first come online,
[23:26.800 --> 23:32.520]  they need to communicate to something hard-coded to figure out what is the network truth, like
[23:32.520 --> 23:39.460]  what are the basic trusted infrastructure that I can use to start getting an idea what is the
[23:39.460 --> 23:48.720]  current block state. A lot of them use protocols like DNS in order to find out what is the first
[23:48.720 --> 23:55.400]  set of nodes that I need to connect to. And if your underlying DNS infrastructure is compromised,
[23:55.400 --> 24:01.860]  they can, again, try to do some kind of eclipse against you. Same with denial of service. If
[24:01.860 --> 24:05.460]  your nodes can never communicate over those protocols, then they will never be able to
[24:05.460 --> 24:11.100]  connect to the network in the first place. BGP routing attacks, those are extremely damaging
[24:12.980 --> 24:17.920]  and something that needs to be considered. I'm putting the severity as high on these,
[24:17.920 --> 24:22.400]  but probability lower, just because it's a lot harder to do a targeted attack to figure out what
[24:22.400 --> 24:29.320]  is the exact node that is owned by your company or by you individual. So, it would most likely be
[24:29.320 --> 24:37.200]  an attack against the entire node network, but it's still probable.
[24:39.900 --> 24:45.540]  So, we can keep going through all of these individual components and try to go through
[24:45.540 --> 24:50.160]  like what are the main threats. Luckily, I went through this exercise over the last few months
[24:50.160 --> 24:58.460]  and I built a no threat model which applies to generic nodes out there. So, if you take a
[24:58.460 --> 25:04.580]  Bitcoin node versus, let's say, Ethereum nodes, they will have all these core issues, all these
[25:04.580 --> 25:09.780]  core threats, plus whatever makes them unique from each other. So, if you're like I mentioned
[25:09.780 --> 25:14.540]  that, for example, OpenEthereum has an extra web server in there. So, you'll need to slap in a web
[25:14.540 --> 25:20.660]  server into this model and consider like what are the threats there as well. Some more terminology
[25:20.660 --> 25:28.060]  so we can interpret the grid that I'm going to show you in a bit. So, stride is a common
[25:28.060 --> 25:33.240]  acronym used for threat modeling. It stands for simply just a variety of bad things you can do,
[25:33.240 --> 25:38.200]  which is spoofing, tampering, repudiation, information disclosure, denial of service,
[25:38.200 --> 25:43.060]  elevation of privilege. It's just a guide to help you think better about like what is the impact
[25:43.060 --> 25:49.060]  there. It is also important to consider the likelihood. How likely is that the threat event
[25:49.060 --> 25:55.060]  is going to occur? How likely is that the threat actor not only has an opportunity but also the
[25:55.060 --> 26:03.080]  expertise to exploit something, the weakness. This will become useful later on as we talk about
[26:03.080 --> 26:10.640]  mitigations and how to prioritize them. So, overall in the threat model, we already went through the
[26:10.640 --> 26:16.020]  11 components I showed you in a few slides ago. The total number of threats that I identified
[26:16.020 --> 26:23.340]  were 14 and the total number of critical threats for a generic node is 5. So, again, depending on
[26:23.340 --> 26:29.600]  how your ecosystem looks like, you know, if you're running a larger set of nodes and you have
[26:29.600 --> 26:34.960]  doctorized images and all that, your threat model will continue growing. This is just to get you
[26:34.960 --> 26:42.420]  started somewhere and provide you foundation. This is just a small glimpse into what the threat
[26:42.420 --> 26:49.140]  model looks like. We have the threat name, so node software generic vulnerabilities, the stride
[26:49.140 --> 26:55.240]  impact, so elevation of privilege, denial of service, severity, probability, description,
[26:55.240 --> 27:00.700]  most importantly, a variety of different mitigations that you can implement to address this threat.
[27:04.720 --> 27:12.760]  So, a key thing that we need to do when considering mitigations and the risk is to
[27:12.760 --> 27:19.580]  consider the cost. The cost, probability, and severity. So, the cost is how much time is it
[27:19.580 --> 27:24.720]  going to take you, how many resources you need to invest to address something. What is the severity
[27:24.720 --> 27:32.660]  and probability of this threat? So, in order to prioritize a variety of threats, I came up with
[27:32.660 --> 27:41.460]  this five-level, I guess, criticality grid where most critical flaws are the ones that are both
[27:41.460 --> 27:50.480]  have high impact, but also high probability of happening. Severe threats are the ones that have
[27:50.480 --> 27:57.900]  high impact, but maybe slightly lower probability of occurring. So, think of your generic web
[27:57.900 --> 28:03.800]  server getting exploited, which is running on your node versus some esoteric block
[28:04.820 --> 28:09.180]  parsing vulnerability that needs to be discovered to be exploited. And all the way to low, where
[28:09.180 --> 28:13.880]  these are the threats which we can address at one point, but only after we're done with
[28:13.880 --> 28:21.860]  the top two or top three, at least. So, of all these mitigations,
[28:21.860 --> 28:26.800]  in the model, I created about 41 mitigations that one could potentially implement. It could
[28:26.800 --> 28:33.240]  be probably more that we can think of, but I'll share the top 10 ones in a bit. Of those
[28:33.240 --> 28:38.920]  mitigations, they're addressing 22 critical threats. What's interesting is that half of those
[28:38.920 --> 28:43.100]  mitigations for critical threats are, in fact, very low cost. This is something that you can
[28:43.100 --> 28:48.980]  implement in a day and will significantly bolster the security stance of your nodes.
[28:49.440 --> 28:55.780]  So, yeah, you can start building fuzzers, you can start looking for every line of code of your nodes
[28:55.780 --> 29:02.880]  to make them secure, and they will make them very secure. But you can also do a bunch of things which
[29:02.880 --> 29:09.120]  are very low cost that will not make your nodes completely secure,
[29:09.120 --> 29:15.060]  but they will also, by the order of magnitude, make them resilient to some of the attacks that I
[29:15.060 --> 29:22.380]  described. So, this is just a different view that in the top bar of the critical vulnerabilities,
[29:22.380 --> 29:29.500]  the ones in light blue, you can see that half of them are basically low-cost solutions that
[29:29.500 --> 29:34.560]  you can implement. And then you have just, you know, if you really want to push it through this,
[29:34.560 --> 29:44.040]  like, it's 99.99% secure, you can continue on that trail, but that will be a little bit more costly.
[29:46.420 --> 29:54.860]  So, with that, let's talk about into node security, top 10 defenses that you can implement.
[30:00.840 --> 30:07.660]  So, the first one, and the most critical one, I think, is secure handling of where is the
[30:07.660 --> 30:14.280]  node software coming from, the repositories, the binary distributions, and so on. One key thing
[30:14.280 --> 30:20.120]  that we do at Coinbase, and something that you could consider as well, is repository pinning.
[30:20.740 --> 30:25.820]  Repositories can get compromised, but if you make sure that you pin to a specific non-Git version
[30:26.350 --> 30:31.840]  that you will work with and you will stick to until you go through some kind of process that,
[30:31.840 --> 30:35.720]  okay, I'm going to switch to the latest release instead of just downloading whatever is the
[30:35.720 --> 30:41.360]  latest one every time you deploy a node, that's something you could consider and could partially
[30:41.360 --> 30:47.360]  mitigate the risk of backdoored repos. Another thing is verification of signatures. So, if you're
[30:47.360 --> 30:52.820]  relying on either source code that you're downloading or binaries that you're downloading,
[30:52.820 --> 30:56.240]  make sure you verify all the signatures that, in the case of Monero,
[30:57.820 --> 31:04.260]  if the binaries were signed and the attackers did not have signed signatures correctly,
[31:04.260 --> 31:08.860]  then you would be able to detect that. In the case of source code, if you can at least verify,
[31:08.860 --> 31:13.220]  like, who is committing to the repositories, is it coming from, am I accessing a website,
[31:13.220 --> 31:17.220]  and it's, you know, the TLS is not breaking, no one is men in the middle of me trying to
[31:17.220 --> 31:23.900]  download the source code. Again, this is something that could make your nodes more secure.
[31:26.160 --> 31:31.780]  We continue on that train of thought, we can also make sure that we build all nodes from source
[31:31.780 --> 31:37.160]  code. I know this is a big ask, but this is something that you could consider, and the
[31:37.160 --> 31:43.860]  reason for this is, again, the Monero example, where only the binaries of Monero were backdoored,
[31:43.860 --> 31:50.220]  the source code remained intact. And the reason why, both could have been compromised, but if you
[31:50.220 --> 31:55.940]  consider how binaries for nodes are built, so there's a software repository with a lot of eyes,
[31:55.940 --> 32:02.300]  and then there could be a flow where one single developer with malware on their machine downloads
[32:02.300 --> 32:07.880]  that source code, builds the library, uploads the binary back on the website, there are a variety
[32:07.880 --> 32:12.800]  of things that could break along the way. Same goes with the CI automation of some sort, which
[32:12.800 --> 32:17.360]  builds binaries automatically. There are a lot more components which could be attacked and
[32:17.360 --> 32:24.320]  compromised, which is why, in case, again, lesson learned from that one incident, is that if you just
[32:24.320 --> 32:29.080]  stick to the source code, it may still be backdoored, but at least you have, like, one point of failure
[32:29.080 --> 32:36.080]  as opposed to a multitude of others, which are required when building binaries. Secure node
[32:36.080 --> 32:42.700]  configuration. So, every single node that I looked at has a whole variety of different levers that you
[32:42.700 --> 32:48.300]  can pull to make them secure. Some things that you should, generically, you could consider is make
[32:48.300 --> 32:54.120]  sure that you can diversify the number of connections that your node can make, and to which
[32:54.120 --> 33:01.080]  nodes it can make, it can connect to. So, an example would be things like Stellar, I believe. They need
[33:01.080 --> 33:07.760]  to be manually configured to distribute which nodes that you connect to. So, make sure that if
[33:07.760 --> 33:14.920]  you configure nodes like that, you select as wide of a node selection as possible. Others, they
[33:14.920 --> 33:19.460]  restrict the number of connections total, so you want to see, like, what is a reasonable number
[33:19.460 --> 33:24.360]  of connections you want to maintain throughout the network. This will help you with Eclipse attacks,
[33:24.360 --> 33:32.240]  making more resilient against forks, and so on. Another important key on configuring nodes
[33:32.240 --> 33:38.560]  securely is locking down your RPC interfaces. So, I believe last year, there was mass scanning of
[33:38.560 --> 33:46.540]  Ethereum nodes that were looking for open RPC interfaces to basically steal private keys. So,
[33:46.540 --> 33:51.140]  lock down your RPC interface. There's no reason why those should be exposed to the rest of the world.
[33:51.140 --> 33:56.840]  So, make sure you configure, you know, whatever IP or interface that RPC is listening on,
[33:56.840 --> 34:00.560]  make sure it's configured to something local to your network or just local host.
[34:02.280 --> 34:08.760]  Restrict node access. So, not everyone needs constant access to your nodes. Not everyone
[34:08.760 --> 34:14.200]  needs to be able to modify arbitrary things within your node. So, at Coinbase, what we do
[34:14.200 --> 34:21.240]  is we create a consensus mechanism with no configuration changes, no nodes, no one can bring
[34:21.240 --> 34:26.120]  up and down nodes just arbitrarily. It requires a consensus of people. The more critical your
[34:26.120 --> 34:32.220]  infrastructure is, the more people it requires a sign-off. So, you can implement it through a
[34:32.220 --> 34:38.340]  sign-off, through some kind of two-factor pings that you can send to people to approve any given
[34:38.340 --> 34:46.440]  action. Define and enforce administrator roles. Again, this is something to restrict to security
[34:46.440 --> 34:51.740]  nodes in larger environments, but you can apply this to individual nodes as well. Make sure that
[34:52.300 --> 34:59.120]  there are some specific roles set up for the node that, you know, you can either configure nodes and
[34:59.120 --> 35:03.720]  deploy them or shut them down, but not everything at the same time. So, you can get very granular
[35:03.720 --> 35:10.120]  there, but it depends, see how appropriate it is for your specific environment. Finally, restrict
[35:10.120 --> 35:18.220]  node access, as in the network traffic node access, both ingress and egress. So, on one side,
[35:18.220 --> 35:22.780]  it makes sense that you restrict your RPC interface already, so make sure those are firewalled off,
[35:22.780 --> 35:27.480]  that you cannot access nodes from anywhere, and then externally, you also have some kind
[35:27.480 --> 35:33.380]  of restriction set up as well. On the other side, nodes could be an excellent points of entry for
[35:33.380 --> 35:37.920]  the attacker who found some kind of vulnerability into the rest of your network. So, make sure that
[35:37.920 --> 35:42.020]  your nodes cannot communicate to the rest of your network as well, because they could be used as
[35:42.020 --> 35:50.460]  stepping stones. Monitor and log node activity. So, this one is probably the most interesting one,
[35:50.460 --> 35:56.380]  and there's a lot of things you can do there. Anytime your node loses internet connectivity,
[35:57.480 --> 36:02.160]  make sure you do some basic verification. Am I getting... is there like something with BGP,
[36:02.160 --> 36:07.980]  or am I getting... is my DNS going down? Am I under attack? Am I getting eclipsed? So,
[36:07.980 --> 36:12.900]  make sure you investigate those. Am I on the right fork? Am I not seeing sufficient traffic?
[36:13.000 --> 36:18.820]  All of that applies. You can do node-specific verifications too, not just generic network
[36:19.780 --> 36:24.920]  monitoring, such as periodically check, like, am I on the right fork? What is the block height?
[36:24.920 --> 36:30.700]  What is the current block hash? Do other nodes in my system, or externally, or on blockchain
[36:30.700 --> 36:37.440]  explorers, do they still see the same thing? So, in case of the recent ETC experiment incident,
[36:37.440 --> 36:41.320]  you would have been able to quickly diagnose, like, oh, okay, so there's something going
[36:41.320 --> 36:48.480]  on with nodes because there's some kind of forking happening. Finally, log all access,
[36:48.480 --> 36:54.720]  all network events. It may not be as essential during the normal operation of node, but when
[36:54.920 --> 36:58.900]  something goes wrong and you need to investigate and figure out what happened, who accessed it,
[36:58.900 --> 37:02.520]  what were the network events that led up to the attack, those logs will be
[37:02.520 --> 37:05.400]  highly instrumental for the investigation.
[37:07.620 --> 37:13.520]  Lockdown-based OS. So, definitely, you cannot build a secure node software if your underlying
[37:13.520 --> 37:20.340]  layer is insecure. If, again, if you want to go after the weakest link, and your node is
[37:20.340 --> 37:26.640]  very well configured and firewalled off, but your underlying OS is some outdated version of Linux,
[37:26.640 --> 37:31.700]  which has, like, exposed ports and they're easily attacked, the attacker is just going to go
[37:31.700 --> 37:36.500]  after that and going to skip all of the other mitigations you built. So, you could do things
[37:36.500 --> 37:41.700]  like doctorize your nodes, run SCLinux or similar toolkits. So, there are a variety of things you
[37:41.700 --> 37:47.120]  can do to lock down the OS, and there are guides for that, but something to consider as well.
[37:48.820 --> 37:54.880]  Hardened node's network connections. So, going off the stack a little bit, not just protecting
[37:54.880 --> 37:59.880]  what connections can be made, or what we're monitoring, make sure that if you configure DNS
[37:59.880 --> 38:05.980]  server for your node, make sure it's a secure one, uses DNSSEC. If you're using ISPs, make sure that
[38:05.980 --> 38:10.360]  whatever BGP connections that it uses are done in a secure fashion. If your node needs to
[38:10.360 --> 38:15.720]  communicate to some kind of HTTP resources, make sure you configure your CAs properly,
[38:15.720 --> 38:21.520]  so you're not getting met in the middle. Denial of service protection is critical. Again, it's
[38:23.360 --> 38:27.860]  potentially hard to determine. I want to attack just one particular node for one particular
[38:28.460 --> 38:32.580]  user or organization, but it's prudent to use something like Cloudflare to
[38:32.580 --> 38:37.280]  have a layer of denial of service protection, so you don't have to worry about that.
[38:38.500 --> 38:45.500]  Node-specific threats. Node and protocol-specific threats. This is a catch-all. Basically,
[38:45.940 --> 38:51.180]  be mindful that all nodes are unique in their own different ways. So, I mentioned before that
[38:51.180 --> 38:55.740]  Stellar, you have to manually configure what are the trusted nodes. You have to manually calculate,
[38:55.740 --> 39:01.000]  like, hey, what is my threshold for... it's a BFT network. What is my threshold that
[39:01.000 --> 39:05.980]  it can get attacked? So, make sure that I'm not going to be vulnerable to that.
[39:05.980 --> 39:14.160]  Other protocols like EOS, they implement debugging modes that you have to watch out for.
[39:14.160 --> 39:19.120]  So, read the config file completely. Make sure you understand what are the implications of all the
[39:19.120 --> 39:26.100]  variety of settings, and have a good secure configuration specific for that node.
[39:28.260 --> 39:32.600]  Verify node functionality before deployment. So, that's a much harder one,
[39:32.600 --> 39:39.860]  because, sure, you pinned the repository. You are pretty confident, okay, so it probably was
[39:39.860 --> 39:45.340]  not modified, but I'm not 100% sure that whatever I downloaded is working as it's
[39:45.340 --> 39:50.500]  supposed to. There are no backdoors there. How do you verify that it still works as intended?
[39:51.180 --> 39:55.120]  Well, ideally, you would run a set of tests, like, hey, if I transmit this transaction,
[39:55.640 --> 39:59.800]  it shows up on the network, or if I see a transaction coming from the network,
[39:59.800 --> 40:06.640]  my account gets properly credited. And that's a big ask to implement for every single node.
[40:06.880 --> 40:14.060]  So, one project which aims to solve this problem is Coinbase Rosetta project, which creates a kind
[40:14.060 --> 40:19.020]  of a middleware for a variety of different nodes to be able to speak the same API. So,
[40:19.020 --> 40:26.820]  instead of implementing a variety of different calls to transmit a transaction, get a current
[40:26.820 --> 40:33.800]  block hash and transactions in it, Rosetta creates a kind of a middleware for a variety
[40:33.800 --> 40:40.120]  of different nodes that you can just use a single API call, and it automatically translates to
[40:40.120 --> 40:45.840]  whatever is specific for that node. Now, what's interesting about the Rosetta project also
[40:45.840 --> 40:52.160]  includes a Rosetta validator, which exercises the node and makes sure that it passes a variety of
[40:52.160 --> 40:59.380]  very specific tests from transaction verification blocks and signing and so on to make sure that it
[40:59.380 --> 41:07.040]  works exactly as intended. Not every project supports Rosetta interface, but for the ones
[41:07.040 --> 41:13.140]  that do, you can implement it in the pipeline to make sure that the nodes that you deploy are acting
[41:13.140 --> 41:21.320]  as expected. And you notice that we're slowly ramping up the difficulty level. This is probably
[41:21.320 --> 41:28.340]  the hardest one. This requires you to examine the source code, examine the alerts that you're
[41:28.340 --> 41:36.880]  getting, and this is the basic security static analysis of node software. So,
[41:36.880 --> 41:40.880]  you may not need to go through the actual source code, but you can observe a variety of different
[41:40.880 --> 41:46.320]  alerts reported by tools like Salus. So, this is another Coinbase tool, which we use internally,
[41:46.320 --> 41:53.400]  and it's an open source tool you can use as well to quickly run through a variety of node
[41:53.400 --> 41:59.360]  source code packages, and it gives you a nice report of what are the high critical vulnerabilities
[41:59.360 --> 42:04.560]  within that. So, you can examine them, just get a general sense. It's hard to say which ones are
[42:04.560 --> 42:07.900]  false positives without actually looking at the source code, but you can see like, hey, if I
[42:07.900 --> 42:13.860]  suddenly see 100 or so, or like this really sneaky report, maybe I should investigate more or report
[42:13.860 --> 42:21.000]  or ask the original developers. In the Salus project, it supports a variety of different
[42:22.600 --> 42:31.440]  languages from Go to Ruby and so on. GoSec, like for if you're just running Ethereum or other
[42:31.440 --> 42:39.260]  Golang-based nodes, so GoSec is excellent. In fact, Salus is running a variety of open source
[42:39.260 --> 42:44.460]  projects, and it just creates a wrapper for them. So, GoSec is one of them. For all the
[42:44.460 --> 42:48.760]  other projects, you have something like Semgrep, so if you have SQL to analyze and so on,
[42:48.760 --> 42:56.460]  so you can use that as well. So, with that, just to summarize, what is our total view of
[42:57.460 --> 43:04.000]  all the different mitigations, top 10 things that you can do to secure your nodes. So,
[43:04.000 --> 43:08.860]  make sure you handle software repositories securely. Make sure that you pin them,
[43:08.860 --> 43:16.320]  that you understand that they can be compromised, and you need to just investigate a little bit
[43:16.320 --> 43:25.040]  more. Build all nodes from source code. So, again, it's just reducing your attack vectors just a
[43:25.040 --> 43:28.600]  little bit more. You just build everything from source. There are a lot of eyes in source code.
[43:28.600 --> 43:33.940]  Who knows what happens with those binaries? Securely configure nodes. So, make sure that
[43:33.940 --> 43:40.200]  you diversify what connections your nodes have. Make sure that they can't be just as easily
[43:40.200 --> 43:48.600]  eclipsed and people can access it and reconfigure them. Restrict node access. Monitor and log node
[43:48.600 --> 43:53.760]  activity. So, you can get, again, very creative with those, what you can do to judge what nodes
[43:53.760 --> 44:00.580]  help. Lock down your base OS. Make sure that you're not running, you're not building a castle
[44:00.580 --> 44:08.060]  on quicksand. Definitely consider node and protocol specific threats. All nodes are unique
[44:08.060 --> 44:13.460]  in a variety of different ways. So, you need to consider those as well. Verify node functionality.
[44:13.460 --> 44:20.500]  So, if your node supports Rosetta project, run Rosetta validator. It will give you this good
[44:20.500 --> 44:26.920]  feeling that, okay, at least for a variety of tests, this still works as expected. And then,
[44:26.920 --> 44:31.780]  if you have time and you have technical expertise, try to run basic static analysis tools
[44:31.780 --> 44:34.720]  just to periodically see, like, are there, like, major
[44:34.720 --> 44:38.220]  low-hanging fruit vulnerabilities that are getting introduced that I can catch,
[44:38.220 --> 44:44.980]  maybe report to original developers. So, with that, thank you so much for your time.
[44:45.920 --> 44:51.480]  I may have not mentioned this. I'm the blockchain threat intelligence newsletter. I list a lot of
[44:51.920 --> 44:57.020]  node incidents and vulnerabilities that are getting discovered. So, subscribe. See if you
[44:57.020 --> 45:03.200]  can catch the latest lessons learned from those vulnerabilities. And feel free to follow me on
[45:03.200 --> 45:10.300]  Twitter. And or feel free to contact me offline if you have any more questions. Thank you.
